Exploiting Arabic Diacritization for High Quality Automatic Annotation

What it is about

Undiacritized Arabic is highly ambiguous: about 12.8 analyses per word out of context. The authors use the diacritics that a text does have to choose among those analyses. MADAMIRA (with the SAMA analyzer) ranks the candidates as usual; then the candidates are filtered by the edit distance between their diacritized form and the input diacritics, and the closest top-ranked one is kept. On a 63K-word newswire text, fully diacritized, the diacritized form rises from 87.8% to 99.1% and the diacritics, lemma, POS and tokenization combined from 86.6% to 97.1%. On two other genres, a children's novel (1,175 words) and a religious text (970 words), the same combination goes from 81.0% to 95.6% and from 72.4% to 92.8%. Partial diacritics help too: keeping only the final diacritic (about 23% of all points) does as well as keeping 80% of them at random, once the edit distance treats missing points as unspecified rather than as absent. Random ranking gives much lower scores than the tagger's ranking, so both steps are needed.

Ideas

• I1. Filter the candidates of a pointed-text analysis by the distance between their vowel points and the input points, as the paper does: after normalizing the variants, keep the analyses closest to the text as written. Test it in the pointed mode of tools/hbo on held-out MACULA chapters, with the tagger ranking alone and with the filter.
• I2. Treat missing points as unspecified, not as absent: change the distance so that a point the text does not show does not count as a disagreement. For partly pointed Hebrew (editions with some points, or rabbinic texts), draw the accuracy-against-point-density curve, as in the paper's Figure 1, and test which point (the final vowel, the dagesh, the shin/sin dot) carries most of the information.
• I3. A normalization table applied before the distance: list the spelling variants that differ only in one sign (for Hebrew: qamats and qamats qatan, sheva as silent or not, dagesh and matres variants) and map each to one form in the distance step. The table is a short rule file in 1. Script and pointing.

For Mova

I1 and I3 go into tools/hbo (pointed mode) and 1. Script and pointing. I2 goes into the unpointed and partly pointed modes of tools/hbo, and its curve is a measure in 13. Biblical Hebrew in Mova: the tools. The paper also shows the cost side: a typist can add points for a small fee, so a pointed-text pilot is cheap. Its data and tools are research-only.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{habash2016exploiting,
  title = {{Exploiting Arabic Diacritization for High Quality Automatic Annotation}},
  author = {Habash, Nizar and Shahrour, Anas and Al-Khalil, Muhamed},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  url = {https://aclanthology.org/L16-1681/}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Abandah, G. A., Graves, A., Al-Shagoor, B., Arabiyat, A., Jamour, F., and Al-Taee, M. (2015). Automatic diacritization of Arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. find
2. Alansary, S. and Nagi, M. (2014). The international corpus of Arabic: Compilation, analysis and evaluation. ANLP 2014, page 8. find
3. Alghamdi, M. and Muzafar, Z. (2007). KACST Arabic diacritizer. In First International Symposium on Computers and Arabic Language, pages 25–28. find
4. Alkuhlani, S., Habash, N., and Roth, R. (2013). Automatic morphological enrichment of a morphologically underspecified treebank. In HLT-NAACL, pages 460–470. find
5. Baghdadi, A. M. i. G. (1987). Majma‘ Al-damanat. Alam Al-Kutub. find
6. Bebah, M., Amine, C., Azzeddine, M., and Abdelhak, L. (2014). Hybrid approaches for automatic vowelization of Arabic texts. arXiv preprint arXiv:1410.2646. ↗
7. Belinkov, Y. and Glass, J. (2015). Arabic diacritization with recurrent neural networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281–2285, Lisbon, Portugal, September. Association for Computational Linguistics. find our card
8. Bouamor, H., Zaghouani, W., Diab, M., Obeid, O., Oflazer, K., Ghoneim, M., and Hawwari, A. (2015). A pilot study on Arabic multi-genre corpus diacritization. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 80–88, Beijing, China, July. Association for Computational Linguistics. find
9. Buckwalter, T. (2004). Buckwalter Arabic Morphological Analyzer Version 2.0. LDC catalog number LDC2004L02, ISBN 1-58563-324-0. find
10. Diab, M., Habash, N., Rambow, O., and Roth, R. (2013). LDC Arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
11. Graff, D., Maamouri, M., Bouziri, B., Krouna, S., Kulick, S., and Buckwalter, T. (2009). Standard Arabic Morphological Analyzer (SAMA) Version 3.1. Linguistic Data Consortium LDC2009E73. find
12. Habash, N. and Rambow, O. (2007). Arabic Diacritization through Full Morphological Tagging. In Proceedings of the 8th Meeting of the North American Chapter of the Association for Computational Linguistics/Human Language Technologies Conference (HLT-NAACL07). find our card
13. Habash, N. and Roth, R. (2009). CATiB: The Columbia Arabic Treebank. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 221–224, Suntec, Singapore. find our card
14. Habash, N., Gabbard, R., Rambow, O., Kulick, S., and Marcus, M. (2007a). Determining case in Arabic: Learning complex linguistic behavior requires complex linguistic features. In Proceeings of the The 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2007), Prague, June. find
15. Habash, N., Soudi, A., and Buckwalter, T. (2007b). On Arabic Transliteration. In A. van den Bosch et al., editors, Arabic Computational Morphology: Knowledge-based and Empirical Methods. Springer. find
16. Habash, N. (2010). Introduction to Arabic Natural Language Processing. Morgan & Claypool Publishers. find
17. Hadj Ameur, M. S., Moulahoum, Y., and Guessoum, A. (2015). Restoration of Arabic diacritics using a multilevel statistical model. In Computer Science and Its Applications, volume 456 of IFIP Advances in Information and Communication Technology, pages 181–192. Springer International Publishing. find
18. Kilany, K. (2013). Wazzat Al-sultan. Hindawi Foundation for Education and Culture. find
19. Maamouri, M. and Cieri, C. (2002). Resources for Natural Language Processing at the Linguistic Data Consortium. In Proceedings of the International Symposium on Processing of Arabic, pages 125–146, Manouba, Tunisia. find
20. Maamouri, M., Bies, A., Buckwalter, T., and Mekki, W. (2004). The Penn Arabic Treebank: Building a Large-Scale Annotated Arabic Corpus. In NEMLAR Conference on Arabic Language Resources and Tools, pages 102–109, Cairo, Egypt. find
21. Maamouri, M., Bies, A., and Kulick, S. (2006). Diacritization: A challenge to Arabic treebank annotation and parsing. In Proceedings of the Conference of the Machine Translation SIG of the British Computer Society. find
22. Maamouri, M., Bies, A., and Kulick, S. (2008). Enhancing the Arabic Treebank: a Collaborative Effort toward New Annotation Guidelines. In European Language Resources Association (ELRA), editor, Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), Marrakech, Morocco, May. find
23. Maamouri, M., Bies, A., and Kulick, S. (2009). Creating a Methodology for Large-Scale Correction of Treebank Annotation: The Case of the Arabic Treebank. In Proceedings of MEDAR International Conference on Arabic Language Resources and Tools, Cairo, Egypt. find
24. Nelken, R. and Shieber, S. (2005). Arabic Diacritization Using Weighted Finite-State Transducers. In Proceedings of the Workshop on Computational Approaches to Semitic Languages at 43rd Meeting of the Association for Computational Linguistics (ACL’05), pages 79–86, Ann Arbor, Michigan. find our card
25. Pasha, A., Al-Badrashiny, M., Kholy, A. E., Eskander, R., Diab, M., Habash, N., Pooleery, M., Rambow, O., and Roth, R. (2014). MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In In Proceedings of LREC, Reykjavik, Iceland. find our card
26. Rashwan, M., Al-Badrashiny, M., Attia, M., and Abdou, S. (2009). A hybrid system for automatic Arabic diacritization. In The 2nd International Conference on Arabic Language Resources and Tools. Citeseer. find
27. Shahrour, A., Khalifa, S., and Habash, N. (2015). Improving Arabic diacritization through syntactic analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1309–1315. find our card
28. Smrž, O. and Hajič, J. (2006). The Other Arabic Treebank: Prague Dependencies and Functions. In Ali Farghaly, editor, Arabic Computational Linguistics: Current Implementations. CSLI Publications. find
29. Vergyri, D. and Kirchhoff, K. (2004). Automatic Diacritization of Arabic for Acoustic Modeling in Speech Recognition. In Ali Farghaly et al., editors, COLING 2004 Workshop on Computational Approaches to Arabic Script-based Languages, pages 66–73, Geneva, Switzerland. find
30. Zitouni, I., Sorensen, J. S., and Sarikaya, R. (2006). Maximum entropy based restoration of Arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 577–584, Sydney, Australia, July. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2016-habash-exploiting-arabic-diacritization-high-quality/