Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation

What it is about

The paper predicts the diacritics (short vowels and gemination marks) of Arabic text character by character. It works on a cleaned version of the Tashkeela corpus, with about 2.3 million words, and it introduces diacritic error rate (DER) and word error rate (WER) definitions that ignore unmarked characters and, optionally, the case ending. Two families are tried. A feed-forward model sees a window of 50 unpointed characters on each side; its embedding variant reaches 4.06% DER (9.38% WER with case ending excluded) against 13.78% DER for the rule-based Mishkal system. A recurrent model (two BiLSTM layers, 512 units, with a block-normalized gradient) reaches 1.34% DER and 2.91% WER when trained with an extra set of classical texts and the Quran; a CRF output layer cut the word error rate further in the no-extra-data setting. Prediction is over 15 classes: no mark, the single marks, and the shadda compounds. The last character is the hardest, because its vowel depends on the part of speech.

Ideas

• I1. Adopt the DER/WER definitions as the scoring rule for any pointing or vocalization output of tools/hbo, reporting both with and without the final letter, so Hebrew numbers can be compared with Arabic ones.
• I2. A character-window classifier as a cheap baseline for Hebrew niqqud: 50 characters each side is more than Hebrew needs, so run a window of 5 to 10 consonants, with the ±1 consonant context and the dagesh and shin-dot classes as the label set, and check it against the pointed MACULA text.
• I3. The final-letter result as a rule: the vowel of a word-final letter depends on its part of speech, so a pointing model for unpointed Hebrew should take the parser's POS (from gparse) as an input for the last consonant only; the paper's CRF-versus-softmax result says to test this on the held-out words rather than assume it.

For Mova

I1 is a measure for tools/hbo (the unpointed mode) and for 1. Script and pointing and 2. Cantillation: the accents (te'amim), where the pointing rules are described. I2 is a baseline to build in tools/hbo before any neural pointing, with a held-out check on MACULA. I3 belongs to The Noun, the Adjective and the Numeral (the final vowel depends on the noun class) and to tools/hbo as an input to the pointer. The Translation over Diacritization idea is not for Mova: it depends on a neural translation model.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-5229

@inproceedings{fadel2019neural,
  title = {{Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation}},
  author = {Fadel, Ali and Tuffaha, Ibraheem and Al-Jawarneh, Bara' and Al-Ayyoub, Mahmoud},
  booktitle = {Proceedings of the 6th Workshop on Asian Translation},
  year = {2019},
  doi = {10.18653/v1/d19-5229},
  url = {https://doi.org/10.18653/v1/d19-5229}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gheith Abandah, Alaa Arabiyat, et al. 2017. Investigating hybrid approaches for arabic text diacritization with recurrent neural networks. In 2017 IEEE Jordan Conference on Applied Electrical Engineering BLEU scores are computed with multi-bleu.perl 11 https://github.com/AliOsm/ translation-over-diacritization John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul):2121–2159. and Computing Technologies (AEECT), pages 1–6. IEEE. ↗
2. Gheith A Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015. Automatic diacritization of arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. find
3. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019. Arabic text diacritization using deep neural networks. In ICCAIS. find
4. Amany Fashwan and Sameh Alansary. 2017. Shakkil: an automatic diacritization system for modern standard arabic texts. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 84– 93. find
5. Rehab Alnefaie and Aqil M Azmi. 2017. Automatic minimal diacritization of arabic texts. Procedia Computer Science, 117:169–174. find
6. Sawsan Alqahtani, Hanan Aldarmaki, and Mona Diab. 2019. Homograph disambiguation through selective diacritic restoration. In Proceedings of the Fourth Arabic Natural Language Processing Workshop, pages 49–59. find
7. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
8. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. find
9. Jeremy Appleyard, Tomas Kocisky, and Phil Blunsom. 2016. Optimizing performance of recurrent neural networks on gpus. arXiv preprint ar-Xiv:1604.01946. find
10. Rajae Moumen, Raddouane Chiheb, Rdouan Faizi, and Abdellatif El Afia. 2018. Evaluation of gated recurrent unit in arabic diacritization. International Journal of Advanced Computer Science and Applications (IJACSA), 9(11). find
11. Aqil M Azmi and Reham S Almajed. 2015. A survey of automatic arabic diacritization techniques. Natural Language Engineering, 21(3):477–495. find
12. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint ar-Xiv:1409.0473. find
13. Hamdy Mubarak, Ahmed Abdelali, Hassan Sajjad, Younes Samih, and Kareem Darwish. 2019. Highly effective arabic diacritization using sequence to sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2390–2395. find
14. Ahmad Barqawi and Taha Zerrouki. 2017. Shakkala, arabic text vocalization. find
15. Mohamed Bebah, Chennoufi Amine, Mazroui Azzeddine, and Lakhouaja Abdelhak. 2014. Hybrid approaches for automatic vowelization of arabic texts. arXiv preprint arXiv:1410.2646. ↗
16. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101. find our card
17. Yonatan Belinkov and James Glass. 2015. Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285. find our card
18. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. ↗
19. Tiago Carneiro, Raul Victor Medeiros Da Nóbrega, Thiago Nepomuceno, Gui-Bin Bian, Victor Hugo C De Albuquerque, and Pedro Pedrosa Reboucas Filho. 2018. Performance analysis of google colaboratory as a tool for accelerating deep learning applications. IEEE Access, 6:61677–61685. find
20. Anas Shahrour, Salam Khalifa, and Nizar Habash. 2015. Improving arabic diacritization through syntactic analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1309–1315. find our card
21. Amine Chennoufi and Azzeddine Mazroui. 2017. Morphological, syntactic and diacritics rules for automatic diacritization of arabic sentences. Journal of King Saud University-Computer and Information Sciences, 29(2):156–163. find
22. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214– 2218. find
23. Adams Wei Yu, Lei Huang, Qihang Lin, Ruslan Salakhutdinov, and Jaime Carbonell. 2017. Block-normalized gradient method: An empirical study for training deep neural network. arXiv preprint ar-Xiv:1707.04822. find
24. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. find our card
25. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of arabic vocalized texts, data for auto-diacritization systems. Data in brief, 11:147. find
26. Imed Zitouni and Ruhi Sarikaya. 2009. Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23(3):257– 276. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2019-fadel-neural-arabic-text-diacritization-state/