A Multitask Learning Approach for Diacritic Restoration

What it is about

Diacritic restoration for Arabic (adding the short vowels to undiacritized text) is usually done by a character-level BiLSTM, which can lose word-level information. The authors train three auxiliary tasks alongside it: word segmentation, POS tagging with Universal Dependencies tags, and syntactic diacritization (the case ending of each word). Each character is concatenated with a word embedding, and the auxiliary outputs feed the main model. On the ATB data (parts 1–3) the full joint model reaches a word error rate of 7.51, against 8.09 for the word-to-character baseline and 8.51 for the character baseline. The gain is largest on syntactic endings, and removing any one auxiliary task hurts. Segmentation on its own slightly hurts.

Ideas

• I1. Word-level context as a feature in the unpointed mode of tools/hbo: concatenate each consonant with a word-level vector (the root and prefix analysis from the segmenter) before vowel prediction, and measure the error rate on out-of-vocabulary words only.
• I2. An ablation table in the paper's style: remove one auxiliary task at a time from the vowel model in tools/hbo, average over several runs, and test the differences. The paper's protocol is a good template.
• I3. A caution to test before use: segmentation alone hurt, because splitting the pieces obscures non-concatenative (root-and-pattern) morphology. Keep segmentation out of the vowel model unless the ablation shows a gain.

For Mova

I1 and I2 go into tools/hbo (unpointed mode and its tests); the out-of-vocabulary measure is kept with The lexicon. I3 is a design note for tools/hbo beside the segmenter tables, linked from Pronouns, particles and prepositions only where prefixes are discussed.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.732

@inproceedings{alqahtani2020multitask,
  title = {{A Multitask Learning Approach for Diacritic Restoration}},
  author = {Alqahtani, Sawsan and Mishra, Ajay and Diab, Mona},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.732},
  url = {https://doi.org/10.18653/v1/2020.acl-main.732}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sawsan Alqahtani and Mona Diab. 2019a. Investigating input and output units in diacritic restoration. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA). IEEE. find
2. Sawsan Alqahtani and Mona Diab. 2019b. Investigating input and output units in diacritic restoration. In 2019 18th IEEE International Conference on Machine Learning and Applications (ICMLA). find
3. Sawsan Alqahtani, Ajay Mishra, and Mona Diab. 2019. Convolutional neural networks for diacritic restoration. In EMNLP. find
4. Sankaranarayanan Ananthakrishnan, Shrikanth Narayanan, and Srinivas Bangalore. 2005. Automatic diacritization of arabic transcripts for automatic speech recognition. In Proceedings of the 4th International Conference on Natural Language Processing, pages 47–54. find
5. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics. find
6. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic diacritization: Stats, rules, and hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. find our card
7. Mona Diab, Mahmoud Ghoneim, and Nizar Habash. 2007. Arabic diacritization in the context of statistical machine translation. In Proceedings of MT-Summit. find
8. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. Ldc arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
9. Mona Diab, Kadri Hacioglu, and Daniel Jurafsky. 2004. Automatic tagging of arabic text: From raw text to base phrase chunks. In Proceedings of HLT-NAACL 2004: Short papers. Association for Computational Linguistics. find
10. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhikers guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 1383–1392. find
11. Nizar Habash, Ryan Gabbard, Owen Rambow, Seth Kulick, and Mitch Marcus. 2007. Determining case in arabic: Learning complex linguistic behavior requires complex linguistic features. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find
12. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2016. A joint many-task model: Growing a neural network for multiple nlp tasks. arXiv preprint arXiv:1611.01587. ↗
13. Ehab W Hermena, Denis Drieghe, Sam Hellmuth, and Simon P Liversedge. 2015. Processing of arabic diacritical marks: Phonological–syntactic disambiguation of homographic verbs and visual crowding effects. Journal of Experimental Psychology: Human Perception and Performance, 41(2):494. find
14. Yasser Hifny. 2018. Hybrid lstm/maxent networks for arabic syntactic diacritics restoration. IEEE Signal Processing Letters, 25(10):1515–1519. find
15. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
16. Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7482–7491. find
17. Iroro Orife. 2018. Attentive sequence-to-sequence learning for diacritic restoration of yor\ub\’a language text. arXiv preprint arXiv:1804.00832. ↗
18. Arfath Pasha, Mohamed Al-Badrashiny, Mona T Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In LREC, volume 14, pages 1094–1101. find our card
19. Mohsen AA Rashwan, Ahmad A Al Sallab, Hazem M Raafat, and Ahmed Rafea. 2015. Deep learning framework with confused sub-set resolution architecture for automatic arabic diacritization. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 23(3):505–516. find
20. Ahmed Said, Mohamed El-Sharqwi, Achraf Chalabi, and Eslam Kamal. 2013. A hybrid approach for arabic diacritization. In International Conference on Application of Natural Language to Information Systems, pages 53–64. Springer. find
21. Nasser Zalmout and Nizar Habash. 2017. Don’t throw those morphological analyzers away just yet: Neural morphological disambiguation for arabic. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704– 713. find our card
22. Nasser Zalmout and Nizar Habash. 2019a. Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling. arXiv preprint arXiv:1910.12702. ↗ our card
23. Nasser Zalmout and Nizar Habash. 2019b. Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging. arXiv preprint arXiv:1910.02267. ↗ our card
24. Imed Zitouni and Ruhi Sarikaya. 2009. Arabic diacritic restoration approach based on maximum entropy models. Computer Speech & Language, 23(3):257– 276. find
25. Imed Zitouni, Jeffrey S Sorensen, and Ruhi Sarikaya. 2006. Maximum entropy based restoration of arabic diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 577–584. Association for Computational Linguistics. find our card
26. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing, 45(11):2673–2681. find
27. Khaled Shaalan, Hitham M Abo Bakr, and Ibrahim Ziedan. 2009. A hybrid approach for building arabic diacritizer. In Proceedings of the EACL 2009 workshop on computational approaches to semitic languages, pages 27–35. Association for Computational Linguistics. find
28. Anas Shahrour, Salam Khalifa, and Nizar Habash. 2015. Improving arabic diacritization through syntactic analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1309–1315. find our card
29. Dima Taji, Nizar Habash, and Daniel Zeman. 2017. Universal dependencies for arabic. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 166–176. find our card
30. Daniel Watson, Nasser Zalmout, and Nizar Habash. 2018. Utilizing character and word embeddings for text normalization with sequence-to-sequence models. arXiv preprint arXiv:1809.01534. ↗
31. JC Wells. 2000. Orthographic diacritics and multilingual computing. Language problems and language planning, 24(3):249–272. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2020-alqahtani-multitask-learning-approach-diacritic-restoration/