Improving Arabic Diacritization with Regularized Decoding and Adversarial Training

What it is about

Diacritization models often treat auto-generated labels (from toolkits such as morphological analysers) as gold, so the toolkit's errors pass into the model. The authors route the auto labels to a separate regularized decoder that shares a character encoder with the main tagger. An adversarial discriminator tries to tell from the shared encoding whether the labels came from gold or from the auto source, and the encoder is pushed toward indifference. Training has two stages: a joint stage, then fine-tuning of the main tagger on gold labels alone with the shared encoder frozen. Tested on ATB and Tashkeela, with BiLSTM and Transformer encoders and with AraBERT and ZEN 2.0 embeddings, the models report lower DER and WER than earlier systems on both datasets, according to the paper's tables. The abstract claims state of the art on both.

Ideas

• I1. Keep teacher labels apart from gold: labels from a large teacher model are noisy, so put them in a separate head that shares the encoder, and train the final tagger on gold only. Measure on held-out gold, not on the release test set.
• I2. A noise-injection control: corrupt a rule-based tagger's output on purpose (for example 10% and 30% of the labels) and check whether the trained model still stays above the rule-only baseline. This is the negative-control style test the paper's setup implies.
• I3. The two-stage schedule (joint training, then gold-only fine-tuning with the shared part frozen) as a training option for the gparse parser or for the pointing model in tools/hbo.

For Mova

I1 belongs to the cycle trainer's design, where teacher labels are used. I2 is a test for the pointed mode of tools/hbo. I3 is an option for gparse and for the tools/hbo pointing model.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.68

@inproceedings{qin2021improving,
  title = {{Improving Arabic Diacritization with Regularized Decoding and Adversarial Training}},
  author = {Qin, Han and Chen, Guimin and Tian, Yuanhe and Song, Yan},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.68},
  url = {https://doi.org/10.18653/v1/2021.acl-short.68}
}

References

42 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gheith Abandah and Asma Abdel-Karim. 2019. Accurate and Fast Recurrent Neural Network Solution for the Automatic Diacritization of Arabic Text. Jordanian Journal of Computers and Information Technology, 6. find
2. Gheith A Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad Jamour, and Majid Al-Taee. 2015a. Automatic Diacritization of Arabic Text using Recurrent Neural Networks. International Journal on Document Analysis and Recognition (IJDAR), 18(2):183–197. find
3. Kareem Darwish, Hamdy Mubarak, and Ahmed Abdelali. 2017. Arabic Diacritization: Stats, Rules, and Hacks. In Proceedings of the Third Arabic Natural Language Processing Workshop, pages 9–17. Association for Computational Linguistics. find our card
4. Gheith A. Abandah, Alex Graves, Balkees Al-Shagoor, Alaa Arabiyat, Fuad T. Jamour, and Majid A. Al-Taee. 2015b. Automatic diacritization of Arabic text using recurrent neural networks. International Journal on Document Analysis and Recognition (IJDAR), 18:183–197. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. find
6. Ahmed Abdelali, Kareem Darwish, Nadir Durrani, and Hamdy Mubarak. 2016. Farasa: A Fast and Furious Segmenter for Arabic. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 11–16, San Diego, California. find our card
7. Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. Ldc arabic treebanks and associated corpora: Data divisions manual. arXiv preprint arXiv:1309.5652. ↗
8. Sawsan Alqahtani, Ajay Mishra, and Mona Diab. 2019. Efficient convolutional neural networks for diacritic restoration. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1442–1448. find
9. Sawsan Alqahtani, Ajay Mishra, and Mona Diab. 2020. A multitask learning approach for diacritic restoration. arXiv preprint arXiv:2006.04016. ↗ our card
10. Wissam Antoun, Fady Baly, and Hazem M. Hajj. 2020. AraBERT: Transformer-based Model for Arabic Language Understanding. ArXiv, abs/2003.00104. ↗
11. Alaa Khaled Radwan Arabiyat. 2015. Automatic Arabic Text Diacritization Using Recurrent Neural Networks. Master’s thesis, The University of Jordan. find
12. Yonatan Belinkov and James Glass. 2015. Arabic Diacritization with Recurrent Neural Networks. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2281– 2285. find our card
13. Drago, Burileanu, Vladimir Popescu, Cristian Negrescu, and Aurelian Dervi. 2008. Automatic Diacritic Restoration for a Tts-based E-mail Reader Application. find
14. Ali Fadel, Ibraheem Tuffaha, Mahmoud Al-Ayyoub, et al. 2019a. Arabic Text Diacritization Using Deep Neural Networks. In 2019 2nd International Conference on Computer Applications & Information Security (ICCAIS), pages 1–7. find
15. Ali Fadel, Ibraheem Tuffaha, Bara’ Al-Jawarneh, and Mahmoud Al-Ayyoub. 2019b. Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation. In Proceedings of the 6th Workshop on Asian Translation, pages 215–225. find our card
16. Nizar Habash and Owen Rambow. 2007. Arabic Diacritization through Full Morphological Tagging. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 53–56, Rochester, New York. find our card
17. Guimin Chen, Yuanhe Tian, and Yan Song. 2020. Joint Aspect Extraction and Sentiment Analysis with Directional Graph Convolutional Networks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 272–279. find
18. Ehab Hermena, Denis Drieghe, Sam Hellmuth, and Simon Liversedge. 2015. Processing of Arabic Diacritical Marks: Phonological-Syntactic Disambiguation of Homographic Verbs and Visual Crowding Effects. Journal of experimental psychology. Human perception and performance, 41. find
19. Guimin Chen, Yuanhe Tian, Yan Song, and Xiang Wan. 2021. Relation Extraction with Type-aware Map Memories of Word Dependencies. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021. find
20. Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The Penn Arabic Treebank : Building a Large-Scale Annotated Arabic Corpus. find
21. Kareem Darwish, Ahmed Abdelali, Hamdy Mubarak, and Mohamed Eldesouki. 2020. Arabic Diacritic Recovery Using a Feature-Rich bilstm Model. ArXiv, abs/2002.01207. ↗
22. Angrosh Mandya, Danushka Bollegala, and Frans Coenen. 2020. Graph Convolution over Multiple Dependency Sub-graphs for Relation Extraction. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6424–6435. find
23. Yuyang Nie, Yuanhe Tian, Yan Song, Xiang Ao, and Xiang Wan. 2020. Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4231–4245. find
24. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1094–1101. find our card
25. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. find
26. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227– 2237, New Orleans, Louisiana. find
27. Anas Shahrour, Salam Khalifa, and Nizar Habash. 2015. Improving Arabic Diacritization through Syntactic Analysis. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1309–1315. find our card
28. Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learning Word Representations with Regularization from Prior Knowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 143–152. find
29. Yan Song and Shuming Shi. 2018. Complementary Learning of Word Embeddings. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4368– 4374. find
30. Yuanhe Tian, Guimin Chen, and Yan Song. 2021a. Aspect-based Sentiment Analysis with Type-aware Graph Convolutional Networks and Layer Ensemble. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2910–2922, Online. find
31. Yuanhe Tian, Guimin Chen, Yan Song, and Xiang Wan. 2021b. Dependency-driven Relation Extraction with Attentive Graph Convolutional Networks. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing. find
32. Yuanhe Tian, Wang Shen, Yan Song, Fei Xia, Min He, and Kenli Li. 2020a. Improving Biomedical Named Entity Recognition with Syntactic Information. BMC Bioinformatics, 21:1471–2105. find
33. Yuanhe Tian, Yan Song, and Fei Xia. 2020b. Joint Chinese Word Segmentation and Part-of-speech Tagging via Multi-channel Attention of Character Ngrams. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2073–2084. find
34. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. find
35. Nasser Zalmout and Nizar Habash. 2017. Don’t throw Those Morphological Analyzers Away Just Yet: Neural Morphological Disambiguation for Arabic. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 704–713, Copenhagen, Denmark. find our card
36. Nasser Zalmout and Nizar Habash. 2019. Adversarial multitask learning for joint multi-feature and multi-dialect morphological modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1775–1786, Florence, Italy. find our card
37. Yan Song, Shuming Shi, and Jing Li. 2018. Joint Learning Embeddings for Chinese Words and Their Components via Ladder Structured Networks. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 4375–4381. find
38. Nasser Zalmout and Nizar Habash. 2020. Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8297–8307, Online. find our card
39. Yan Song, Tong Zhang, Yonggang Wang, and Kai-Fu Lee. 2021. ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders. arXiv preprint arXiv:2105.01279. ↗
40. Taha Zerrouki and Amar Balla. 2017. Tashkeela: Novel corpus of Arabic vocalized texts, data for auto-diacritization systems. Data in Brief, 11:147 – 151. find
41. Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. 2020. Dependency Graph Enhanced Dual-transformer Structure for Aspect-based Sentiment Classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6578–6588. find
42. Imed Zitouni, Jeffrey S. Sorensen, and Ruhi Sarikaya. 2006. Maximum Entropy Based Restoration of Arabic Diacritics. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 577–584. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/acl-2021-qin-improving-arabic-diacritization-regularized-decoding/