Arabic Diacritization: Stats, Rules, and Hacks

What it is about

The diacritizer works in two cascaded steps. First it restores the vowels of the word core (the word without its case ending): a word-level bigram model with a Viterbi decoder, backed off to stem dictionaries, stem templates derived from roots (computed with Farasa), a transliteration-mined model for foreign names (a CRF trained on 500 names), and a dictionary of words that have one dominant diacritized form (at least 10 occurrences, more than 90% of them the same). Second it guesses the case ending with an SVM ranker over lexical, morphological and POS features, with filtering rules (a verb restricts the case to a few values, a dual suffix to one value, a DET prefix forbids tanween, a template restricts the endings seen in training). The corpus is a commercial diacritized corpus of 9.7 million tokens. On 70 WikiNews articles (about 18,300 words), the full system reaches 3.29% word error without case endings and 12.76% with them. MADAMIRA gets 6.73% and 19.02% on the same two measures, and Belinkov and Glass 14.87% on the core words and 30.50% with case endings. The system is written in Java, free for research, and needs 2.2 GB of memory.

Ideas

• I1. A back-off ladder for unpointed words in tools/hbo: a fully pointed form first, then the stem and its vowels, then the root-and-pattern template, then a default. Each rung is a table in Rust, and the paper's table of gains per rung is the template for measuring each rung on a held-out Hebrew set.
• I2. Rules that filter case-like candidates before ranking: a construct noun cannot carry the article, a word with the article takes no construct ending, and a verb restricts the final vowel. For Hebrew the same logic restricts the final vowel of a noun in the construct state and forbids the article on construct heads. Put these as filters on candidate vowel patterns, with the count of how many candidates each rule removes.
• I3. A transliteration check for foreign names: a name written with English vowel letters implies its vowels in Arabic script. For Hebrew names with matres lectionis (vav, yod), a rule of this kind can guess the vowels. The paper measured it on a 200-name sample (79% correct), which is the bar a Hebrew version should meet before it is used.

For Mova

I1 goes into the unpointed mode of tools/hbo, and the table of gains per rung becomes its measurement. I2 goes into The Noun, the Adjective and the Numeral (the construct state and the article) as candidate filters. I3 goes into 3. Phonology and the named-entity handling of tools/hbo, measured on a 200-name sample before use.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-1302

@inproceedings{darwish2017arabic,
  title = {{Arabic Diacritization: Stats, Rules, and Hacks}},
  author = {Darwish, Kareem and Mubarak, Hamdy and Abdelali, Ahmed},
  booktitle = {Proceedings of the Third Arabic Natural Language Processing Workshop},
  year = {2017},
  doi = {10.18653/v1/w17-1302},
  url = {https://doi.org/10.18653/v1/w17-1302}
}

References

21 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions, pages 177–180. Association for Computational Linguistics. find
2. Gheith A Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. find
3. Aqil M Azmi and Reham S Almajed. 2015. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21(03):477–495. find
4. Taku Kudo. 2005. Crf++: Yet another crf toolkit. Software available at http://crfpp.sourceforge. net. ↗
5. Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine, and Lakhouaja Abdelhak. 2014. Hybrid approaches for automatic vowelization of arabic texts. arXiv preprint arXiv:1410.2646. ↗
6. Bebah Mohamed Ould Abdallahi Ould, Abdelouafi Meziane, Azzeddine Mazroui, and Abdelhak Lakhouaja. 2011. Alkhalil morphosys. In 7th International Computing Conference in Arabic, pages 66–73, Riyadh, Saudi Arabia. find
7. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285, Lisbon, Portugal. find our card
8. Rani Nelken and Stuart M Shieber. 2005. Arabic diacritization using weighted finite-state transducers. In Proceedings of the ACL Workshop on Computational Approaches to Semitic Languages, pages 79– 86. Association for Computational Linguistics. find our card
9. Tim Buckwalter. 2004. Buckwalter arabic morphological analyzer version 2.0. ldc catalog number ldc2004l02, isbn 1-58563-324-0. find
10. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan M Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC-2014, Reykjavik, Iceland. find our card
11. Andy Cole, David Graff, and Kevin Walker. 2001. Arabic newswire part 1 corpus (1-58563-190-6). Linguistic Data Consortium (LDC). find
12. Kareem Darwish and Hamdy Mubarak. 2016. Farasa: A new fast and accurate arabic word segmenter. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France, may. European Language Resources Association (ELRA). find
13. Mohsen Rashwan, Mohammad Al-Badrashiny, Mohamed Attia, and Sherif Abdou. 2009. A hybrid system for automatic arabic diacritization. In The 2nd International Conference on Arabic Language Resources and Tools, pages 54–60. find
14. Ali El-Kahky, Kareem Darwish, Ahmed Saad Aldein, Mohamed Abd El-Wahab, Ahmed Hefny, and Waleed Ammar. 2011. Improved transliteration mining using graph reinforcement. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1384–1393. find
15. Mohsen Rashwan, Ahmad Al Sallab, M. Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. In IEEE Transactions on Audio, Speech, and Language Processing, pages 505–516. find
16. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to Semitic languages, pages 1–7. Association for Computational Linguistics. find our card
17. Dimitra Vergyri and Katrin Kirchhoff. 2004. Automatic diacritization of arabic for acoustic modeling in speech recognition. In Proceedings of the workshop on computational approaches to Arabic script-based languages, COLING’04, pages 66– 73, Geneva, Switzerland. Association for Computational Linguistics. find
18. Nizar Habash and Owen Rambow. 2007. Arabic diacritization through full morphological tagging. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 53–56. Association for Computational Linguistics. ↗ our card
19. Imed Zitouni, Jeffrey S Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 577–584. Association for Computational Linguistics. find our card
20. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. Mada+ tokan: A toolkit for arabic tokenization, diacritization, morphological disambiguation, pos tagging, stemming and lemmatization. In Proceedings of the 2nd International Conference on Arabic Language Resources and Tools (MEDAR), Cairo, Egypt, pages 102–109. find
21. Thorsten Joachims. 2006. Training linear svms in linear time. In ACM SIGKDD-2006, pages 217–226. ACM. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2017-darwish-arabic-diacritization-stats-rules-hacks/