Mova0.5 pre-release⌕
☰
home / docs / papers / Semitic languages / An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purposes

An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purposes

What it is about

The authors build a rule-based Arabic lemmatizer for information retrieval, without statistics. It starts from the open-source Khoja stemmer, which has a root list of about 3,800 tri-literal and 900 quad-literal roots, and adds a closed-class word list (346 words in 16 groups), a proper-noun list, a list of 61 patterns, affix classes that mark a word as noun or verb, and a dictionary of 943 common verbs whose patterns are ambiguous. A second phase writes the lemma: the perfective third-person masculine singular for verbs, the singular indefinite form for nouns, with a dictionary only for ambiguous broken plurals. On 32,500 words from 50 technical documents, used to tune the rules, it scores 94.8% POS accuracy; on documents it had not seen, 89.15%, against 76.7% for the Stanford Arabic tagger. The ambiguity analysis and the unseen-document figure are the parts worth reusing; the 94.8% comes from the tuned set.

Ideas

For Mova

I1 and I2 are rule tables for tools/hbo and the lexicon in The lexicon. I3 is a lemma dictionary for The Noun, the Adjective and the Numeral. Mova's measurements must follow the paper's lesson: the 89% unseen-document result counts, the 94.8% tuned result does not.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1203.3584

@misc{elshishtawy2012accurate,
  title = {{An Accurate Arabic Root-Based Lemmatizer for Information Retrieval Purposes}},
  author = {El-Shishtawy, Tarek and El-Ghannam, Fatma},
  year = {2012},
  eprint = {1203.3584},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1203.3584}
}

References

29 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Al-Jlayl, M., Frieder O., 2002 . "On Arabic Search: Improving the Retrieval Effectiveness via Light Stemming Approach". In Proceedings of the 11th ACM International Conference on Information and Knowledge Management, Illinois Institute of Technology, New York: ACM Press, pp. 340-347. find
  2. Al-Kharashi L., and M.Evens "Comparing words, stems, and roots as index terms", in an Arabic information retrieval system. JASIS, 45 (8), 1994, pp. 548–560. find
  3. Al-Shammari E., and J. Lin, "A Novel Arabic Lemmatization Algorithm in Proceedings of the second workshop on Analytics for noisy unstructured text data, ACM, 2008. find
  4. Al-Sulaiti L., and E.Atwell, "Extending the Corpus of Contemporary Arabic". in Proceedings of Corpus Linguistics conference, University of Birmingham, UK, 2005. find
  5. Attia M., "An Ambiguity-Controlled Morphological Analyzer for Modern Standard Arabic Modeling Finite State Networks", in The Challenge of Arabic for NLP/MT Conference. The British Computer Society, 2006. find
  6. Beesley, K. "Arabic finite-state morphological analysis and generation" ,In COLING-96: Proceedings of the 16th international, 1996. pp. 89-94. ]7] Buckwalter,T. Qamus: Arabic lexicography. http://members.aol.com/ArabicLexicons/ ↗
  7. Darwish, K., Doermann, D., Jones, R., Oard, D., and Rautiainen, M. , 2001. "TREC-10 experiments at Maryland: CLIR and video" In TREC 2001. Gaithersburg: NIST. find
  8. Dichy J. “On lemmatization in Arabic. A formal definition of the Arabic entries of multilingual lexical databases”. ACL 39th Annual Meeting. Workshop on Arabic Language Processing; Status and Prospect, Toulouse, 2001, 23-30. find
  9. Dichy J, and A.Fargaly, "Roots & Patterns vs. Stems plus Grammar-Lexis Specifications: on what basis should a multilingual lexical database centred on Arabic be built?, Proceedings of the MTSummit, New-Orleans,2003. find
  10. El Isbihani A., K.Shahram, O. Hermann, " Morphosyntactic Arabic Preprocessing for Arabic-to-English Statistical Machine Translation" in Proceedings of the Workshop on Statistical Machine Translation, pages 15–2 New York City, June 2006. find
  11. El-Shishtawy T. and. Al-Sammak A. "Arabic Keyphrase Extraction using Linguistic knowledge and Machine Learning Techniques", Proceedings of the Second International Conference on Arabic Language Resources and Tools, The MEDAR Consortium, Cairo, Egypt. 2009. find
  12. Fedaghi A, Al-Anzi, F. S.,. "A new algorithm to generate Arabic root-pattern forms". In Proceedings of the 11th national computer conference. King Fahd University of Petroleum & Minerals, Dhahran, Saudi Arabia, 1989,pp. 391-400. find
  13. Habash N, Arabic Morphological Representations for Machine Translation in Representations of Arabic Morphology, Soudi A., Van Bosch A. (Ed), Neumann G. (Ed) (2007). find
  14. Habash, N, A.Soudi, and T.Buckwalter, On Arabic Transliteration.: Arabic Computational Morphology-Knowledge-based and Empirical Methods. Van Bosch and A. Soudi. 2007. find
  15. Hajic J., "Morphological tagging: Data vs. dictionaries". in 1st Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’00), Seattle, WA, 2000. find
  16. Hammouda F., and A.Almarimi, "Heuristic Lemmatization for Arabic Texts Indexation and Classification" in Journal of Computer Science 6 (6): 660-665, 2010. find
  17. Ibrahim S. and N. Habash, "Automatic Extraction of Lemma-based Bilingual Dictionaries for Morphologically Rich Languages". in The Third Workshop on Computational Approaches to Arabic Script-based Languages at the Machine Translation Summit XII Ottawa, Ontario, Canada, 2009. find
  18. Khemakhem I., S.Jamoussi, A.Hamadou "Arabic morphosyntactic feature disambiguation in a translation context" Proceedings of SSST-4, Fourth Workshop on Syntax and Structure in Statistical Translation, pages 61–65, Beijing, 2010. find
  19. Khoja, S., 1999. "Stemming Arabic Text". Lancaster, U.K., Computing Department, Lancaster University. www.comp.lancs.uk/computing/users/khoj aJstemmer.ps. find
  20. Kibort A., "What are morphosyntactic features." http://www.surrey.ac.uk/lis/smg/morphosyntacticfeatures.htm 2007 ↗
  21. Larkey L., and I..Ballesteros, "Improving stemming for Arabic information retrieval: light stemming and co-occurrence analysis", Proceedings of the 25th annual international ACM SIGIR conference, 2002. find
  22. Löfberg L., LArcher, D.Piao, S.Rayson, P.Mcenery, T,Varantola, K.Juntunen "Porting an English semantic tagger to the Finnish language", In: Proceedings of the Corpus Linguistics 2003. find
  23. Manning C.. "Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics? In Alexander Gelbukh (ed.), Computational Linguistics and Intelligent Text Processing, 12th International Conference, Proceedings, Part I. Lecture Notes in Computer Science 6608, Springer, 2011 find our card
  24. Pakray P., A. Gelbukh and S.Bandyopadhyay, "A Hybrid Textual Entailment System using Lexical and Syntactic Features", In The 9th IEEE International Conference on Cognitive Informatics, (ICCI 2010), pp. 291-295. 2010. find
  25. Ryan R, O.Rambow, N.Habash, M.Diab, and C.Rud , "Morphological Tagging, Diacritization, and Lemmatization Using Lexeme Models and Feature Ranking" in Proceedings of ACL-08: HLT, pages 117–120, Columbus, Ohio, USA, June 2008. find our card
  26. Sawalha M., and E.Atwell, "Comparative Evaluation of Arabic Language Morphological Analysers and Stemmers", Coling 2008: Companion volume – Posters and Demonstrations, Manchester, 2008, pages 107–110. find
  27. Stanford Natural Language Processing Group, Arabic-accurate.tagger, http://nlp.stanford.edu/software/tagger.shtml, 2011. ↗
  28. Toutanova K., and C.Manning, "Enriching the Knowledge Sources used in a Maximum Entropy Part-of-Speech tagger". In: Joint SIGDAT Conference on Empirical Methods in NLP and Very Large Corpora (EMNLP/VLC-2000) , Hong Kong 2000. َةيبرعلاَةغللاَدعاوقَصخلمَ"،َةمجرتلاوَفيلأتللَيملعلاَبتكملاََةمعنَداؤف َ [31] َ ةعبطلا 31 ، َ 3792 ." َدمحاَ خيشلاَ ، 3171 ."فرصلاَ نفَ ىفَ فرعل ا َ اذشَ "،ةيفاقثلاَ بتكلاَ ةسسؤم find
  29. َ . يولامحلا Tarek El-Shishtawy is a Professor assistant at Faculty of Engineering, Benha University, Egypt. He participated in many Arabic computational Linguistic projects. Large Scale Arabic annotated Corpus, 1995, was one of important projects for Egyptian Computer Society, and Academy of Scientific Research and Technology, He has many publications in Arabic Corpus, machine translation, Text, and data Mining. Fatma El-Ghannam is a researcher assistance at Electronics Research Institute – Cairo, Egypt. She has great research interests in Arabic language generation and analysis. Currently, she's preparing for a Ph.D. degree in NLP. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/langstream-2012-el-shishtawy-accurate-arabic-root-based-lemmatizer/