Build Fast and Accurate Lemmatization for Arabic

What it is about

Arabic lemmatization differs from surface stemming: it must restore the citation form, which can require changing stem patterns, not only cutting affixes. The author builds a fast open-source Java lemmatizer. For each undiacritized word it takes the most frequent diacritized form from a 9.7-million-word corpus, about 73% MSA, and picks the Buckwalter analysis whose diacritized form matches best. Lemma order is fixed by corpus usage where analyses tie. A new test set of 70 WikiNews articles (18,300 words) is lemmatized by an expert and scored on undiacritized lemmas: 73.68% for the Farasa surface stemmer, 96.61% for MADAMIRA and 97.32% for the new system. The paper's text claims +7% relative over MADAMIRA, but the table gap is under one point. Speed: 7.4 million words in 2 minutes, against 2.5 hours for MADAMIRA. The test set is NC, so it is for measurement only.

Ideas

• I1. For unpointed Hebrew, in tools/hbo unpointed mode: when a consonantal form has several analyses, prefer the one whose pointed form is most frequent in a pointed corpus (MACULA gives the pointed forms). It is table-based and needs no context, as the paper does. Where: The lexicon.
• I2. Order lemmas by usage in the lexicon, not in the code: where two analyses share a form (the paper's car/walker case), the lemma with more corpus occurrences comes first, and the order is stored in macula dict.
• I3. Report lemma accuracy on undiacritized text, with an error table in the paper's style (POS error, named-entity segmentation error), as a test for tools/hbo. Build the set from our own open data; do not carry the NC set into a release.

For Mova

I1 goes into tools/hbo (unpointed mode) and The lexicon. I2 goes into macula dict. I3 is the test for tools/hbo.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{mubarak2018build,
  title = {{Build Fast and Accurate Lemmatization for Arabic}},
  author = {Mubarak, Hamdy},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  url = {https://aclanthology.org/L18-1181/}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abdelali, A., Darwish, K., Durrani, N., and Mubarak, H. (2016). Farasa: A fast and furious segmenter for arabic. In HLT-NAACL Demos, pages 11–16. find our card
2. Beesley, K. (1996). Arabic finite-state morphological analysis and generation. In In COLING-96: Proceedings of the 16th international, pages 89–94. find
3. Buckwalter, T. (2002). Arabic finite-state morphological analysis and generation. In http://members.aol.com/ArabicLexicons/. Figure 3: Summary of lemmatization algorithm Figure 4: Lemmatization online demo (part of Farasa Arabic NLP tools) ↗
4. Darwish, K. and Mubarak, H. (2016). Farasa: A new fast and accurate arabic word segmenter. In LREC. find
5. Darwish, K., Mubarak, H., and Abdelali, A. (2017). Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. find our card
6. Dichy, J. and Fargaly, A. (2003). Roots and patterns vs. stems plus grammar-lexis specifications: on what basis should a multilingual lexical database centred on arabic be built? In Proceedings of the MTSummit, New-Orleans. find
7. El-Shishtawy, T. and Al-Sammak, A. (2009). Arabic keyphrase extraction using linguistic knowledge and machine learning techniques. In Proceedings of the Second International Conference on Arabic Language Resources and Tools, The MEDAR Consortium. find
8. El-Shishtawy, T. and El-Ghannam, F. (2012). An accurate arabic root-based lemmatizer for information retrieval purposes. arXiv preprint arXiv:1203.3584. ↗ our card
9. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard arabic morphological analyzer (sama) version 3.1. In Linguistic Data Consortium LDC2009E73. find
10. Khoja, S. (1999). Stemming arabic text. In Computing Department, Lancaster University. find
11. Pasha, A., Al-Badrashiny, M., Diab, M., El Kholy, A., Eskander, R., Habash, N., Pooleery, M., Rambow, O., and Roth, R. M. (2014). Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of Arabic. Proc. LREC. find our card
12. Plisson, J., Lavrac, N., and Mladenic, M. (2004). A rule based approach to word lemmatization. In researchgate.net. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2018-mubarak-build-fast-accurate-lemmatization-arabic/