Arabic Diacritization with Recurrent Neural Networks

What it is about

The paper asks whether Arabic diacritics (short vowels, case endings, gemination) can be restored from diacritized text alone, without a morphological analyser or a tagger. The authors treat diacritization as sequence labelling: each letter gets a label for zero, one or two diacritics, and a character-level recurrent network (LSTM, bidirectional, up to three layers) predicts it from a window of five letters. Data is the Penn Arabic Treebank split used by Zitouni and Sarikaya. Deeper bidirectional LSTMs do best on the development set. On the test split the three-layer model beats the lexical MaxEnt baseline by 3.25 points of diacritic error rate (a 40% error reduction) and beats their best model that used a segmenter and tagger. The main errors are confusions among the short vowels a, i, u and the empty label; the proper name Tueini is missed by all models.

Ideas

• I1. A no-network reference for the unpointed mode of tools/hbo: a lexical maximum-entropy or count model over a five-letter window (as their MaxEnt-lexical row), trained on the same pointed text, so that any neural or rule-based gain is measured against a cheap, transparent baseline.
• I2. Their error analysis as a standing test: a confusion matrix of predicted against gold vowel classes (a/i/u/none) for the Hebrew vowel signs, reported on the held-out set, to show which vowel classes the model confuses.
• I3. Compose the label from a vowel and a consonant mark (their gemination label is a combination of a diacritic with a marker). For Hebrew, a dagesh-plus-vowel label set could let one tag carry both marks instead of separate decisions.

For Mova

I1 and I2 are measures for tools/hbo (unpointed mode): the baseline and the confusion table are tests, not new rules. I3 goes into Phonology of Biblical Hebrew as a label design for pointed Hebrew. The network itself is not used: Mova's SLM stays rule- and table-based.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1274

@inproceedings{belinkov2015arabic,
  title = {{Arabic Diacritization with Recurrent Neural Networks}},
  author = {Belinkov, Yonatan and Glass, James},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1274},
  url = {https://doi.org/10.18653/v1/d15-1274}
}

References

14 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tuka Al Hanai and James Glass. 2014. Lexical Modeling for Arabic ASR: A Systematic Approach. In Proceedings of INTERSPEECH. find
2. Ahmad Al Sallab, Mohsen Rashwan, Hazem M. Raafat, and Ahmed Rafea. 2014. Automatic Arabic diacritics restoration based on deep nets. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP). find
3. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Proceedings of ICASSP. find
4. Nizar Habash and Owen Rambow. 2007. Arabic Diacritization through Full Morphological Tagging. In Proceedings of HLT-NAACL. find our card
5. Nizar Habash, Abdelhadi Soudi, and Timothy Buckwalter. 2007. On Arabic Transliteration. In Abdelhadi Soudi, Günter Neumann, and Antal Van den Bosch, editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods, pages 15–22. Springer. find
6. Nizar Habash, Owen Rambow, and Ryan Roth. 2009. MADA+TOKAN: A Toolkit for Arabic Tokenization, Diacritization, Morphological Disambiguation, POS Tagging, Stemming and Lemmatization. In Proceedings of the Second International Conference on Arabic Language Resources and Tools. find
7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation, 9(8):1735–1780. find
8. Mohamed Maamouri, Ann Bies, and Seth Kulick. 2006. Diacritization: A challenge to Arabic treebank annotation and parsing. In Proceedings of the British Computer Society Arabic NLP/MT Conference. find
9. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR. find
10. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013b. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS. find
11. M.A.A. Rashwan, M.A.S.A.A. Al-Badrashiny, M. Attia, S.M. Abdou, and A. Rafea. 2011. A Stochastic Arabic Diacritizer Based on a Hybrid of Factorized and Unfactorized Textual Features. IEEE Transactions on Audio, Speech, and Language Processing, 19(1):166–175, Jan. find
12. Felix Weninger, Johannes Bergmann, and Björn Schuller. 2015. Introducing CURRENNT: The Munich Open-Source CUDA RecurREnt Neural Network Toolkit. JMLR, 16:547–551. find
13. Imed Zitouni and Ruhi Sarikaya. 2009. Arabic Diacritic Restoration Approach Based on Maximum Entropy Models. Comput. Speech Lang., 23(3). find
14. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum Entropy Based Restoration of Arabic Diacritics. In Proceedings of COLING and ACL. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2015-belinkov-arabic-diacritization-recurrent-neural-networks/